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Abstract. The value 1 problem is a decision problem for probabilistic automata over 
finite words: given a probabilistic automaton, are there words accepted with probability 
arbitrarily close to 1? This problem was proved undecidable recently; to overcome this, 
several classes of probabilistic automata of different nature were proposed, for which the 
value 1 problem has been shown decidable. In this paper, we introduce yet another class 
of probabilistic automata, called leaktight automata, which strictly subsumes all classes of 
probabilistic automata whose value 1 problem is known to be decidable. 

We prove that for leaktight automata, the value 1 problem is decidable (in fact, PSPACE- 
complete) by constructing a saturation algorithm based on the computation of a monoid 
abstracting the behaviours of the automaton. We rely on algebraic techniques developed 
by Simon to prove that this abstraction is complete. Furthermore, we adapt this saturation 
algorithm to decide whether an automaton is leaktight. 

Finally, we show a reduction allowing to extend our decidability results from finite 
words to infinite ones, implying that the value 1 problem for probabilistic leaktight parity 
automata is decidable. 
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Introduction 

Probabilistic automata. Rabin invented a very simple yet powerful model of probabilistic 
machine called probabilistic automata, which, quoting Rabin, “are a generalization of finite 
deterministic automata” |Rab63| . A probabilistic automaton has a finite set of states and 
reads input words from a finite alphabet. The computation starts from the initial state and 
consists in reading the input word sequentially; the state is updated according to transition 
probabilities determined by the current state and the input letter. The probability to accept 
a finite input word is the probability that the computation ends in one of the final states. 

Probabilistic automata, and more generally partially observable Markov decision pro¬ 
cesses and stochastic games, are a widely studied model of probabilistic machines used in 
many fields like software verification [BBG121ICDHR07] , image processing jCK97] , compu¬ 
tational biology [DEKM99j and speech processing IMoli97| . As a consequence, it is crucial 
to understand which decision problems are algorithmically tractable for probabilistic au¬ 
tomata. From a language-theoretic perspective, several algorithmic properties of probabilis¬ 
tic automata are known: while language emptiness is undecidable |Pa,z71 1 lBer74l 1(101 f)j . 
functional equivalence is decidable jSchfill ITze92j as well as other properties [ CMRR08] . 

Our initial motivation for this work comes from control and game theory: we aim 
at solving algorithmic questions about partially observable Markov decision processes and 
stochastic games. For this reason, we consider probabilistic automata as machines controlled 
by a blind controller, who is in charge of choosing the sequence of input letters in order to 
maximize the acceptance probability. While in a fully observable Markov decision process 
the controller can observe the current state of the process to choose adequately the next 
input letter, a blind controller does not observe anything and its choice depends only on 
the number of letters already chosen. In other words, the strategy of a blind controller is 
an input word of the automaton. 

The value of a probabilistic automaton. With this game-theoretic interpretation 
in mind, we define the value of a probabilistic automaton as the supremum acceptance 
probability over all input words, and we would like to compute this value. Unfortunately, 
as a consequence of Paz undecidability result, the value of an automaton is not computable 
in general. However, the following decision problem was conjectured by Bertoni |Ber74j to 
be decidable: 

Value 1 problem: Given a probabilistic automaton, does it have value 1? In other 
words are there input words whose acceptance probability is arbitrarily close to 1? 

Recently, the second and fourth authors of the present paper proved that the value 1 
problem is undecidable [GOlflj . 

Our result. We introduce a new class of probabilistic automata, called leaktight au¬ 
tomata , for which the value 1 problem is decidable. This subclass strictly subsumes all 
known subclasses of probabilistic automata sharing this decidability property and has good 
closure properties. Our algorithm to decide the value 1 problem computes in polynomial 
space a finite monoid whose elements are directed graphs and checks whether it contains a 
certain type of elements that are value 1 witnesses. 

Related works. Introducing subclasses of probabilistic automata to cope with unde¬ 
cidability results has been a fruitful and lively topic recently. We discuss some of them 
here. 
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The first subclass which was introduced specifically to decide the value 1 problem are the 
fl-acyclic automata jGOlOj . Later on, Chatterjee and Tracol [ICT12] introduced structurally 
simple automata, which are probabilistic automata satisfying a structural property (related 
to the decomposition-separation theorem from probability theory), and proved that the 
value 1 problem is decidable for structurally simple automata. At the same time, a subset 
of the authors introduced leaktight automata, and proved a similar result. As we shall 
see, both ((-acyclic and structurally simple automata are leaktight, hence our results extend 
both |G()10j and |GT12| . 

Quite recently, Chadha, Sistla and Viswanathan introduced the subclass of hierarchical 
automata [CSVllj . and showed that over infinite words, they recognize exactly the class 
of w-regular languages. As we shall see, hierarchical automata are leaktight, hence as a 
consequence of our result, the value 1 problem is decidable for hierarchical automata. 

Proof techniques. Our proof techniques totally depart from the ones used in |GSV111 
IGT121 iGOlOj . We make use of algebraic techniques and in particular Simon’s factorization 
forest theorem, which was used successfully to prove the decidability of the boundedness 
problem for distance automata [Sim94| . and extended models as desert automata and B- 
automata (KirOfil ICkdM] 

Outline. Basic definitions are given in Section [T| 

In Section [2] we introduce the Markov monoid and the Markov monoid algorithm for the 
value 1 problem; since the problem is in general undecidable, the algorithm is incomplete: 
a positive answer implies that the automaton has value 1, but a negative answer gives no 
guarantee. 

In Section [3] we define the class of leaktight automata and show that the leaktight 
property is a sufficient condition for this algorithm to be complete; in particular, this 
implies that the value 1 problem is decidable for leaktight automata. 

In Section 01 we show that the Markov monoid algorithm runs in polynomial space, and 
obtain as a corollary that the value 1 problem for leaktight automata is PSPACE-complete. 
Furthermore, we extend the Markov monoid algorithm to check at the same time whether 
an automaton is leaktight and whether in such case it has value 1. 

In Section[5l we further investigate the class of leaktight automata: we provide examples 
of leaktight automata and show that all subclasses of probabilistic automata whose value 1 
problem is known to be decidable are leaktight. 

In Section [6j we give a general theorem allowing to extend the decidability results from 
finite words to infinite words. 


1. Definitions 

1.1. Probabilistic automata. We fix A a finite alphabet. A (finite) word u is a (possibly 
empty) sequence of letters u = a^ai ■ ■ ■ a n _i, the set of finite words is denoted by A*. For 
i < j we denote by u[i,j] the subword a* • • • Oj-i, and u <p = u[0,p] = a^ai ■ ■ ■ a p -\. 

Let Q be a finite set of states. A probability distribution over Q is a function 6 : Q — > 
[0,1] such that = 1; we often see 5 as a row vector of size \Q\. We denote by 

3 ‘ q +1' q' the distribution that picks q with probability ^ and q' with probability |, and by 
q the trivial distribution picking q with probability 1. For a subset R of states, the uniform 
distribution over R picks each state in R with probability ^. The support of a distribution 
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5 is the set of states picked with positive probability, i.e. Supp(<5) = {q £ Q | S(q) > 0}. 
Finally, the set of probability distributions over Q is V(Q). 

Definition 1.1 (Probabilistic automaton). A tuple A = (Q, qo, A, F) represents a proba¬ 
bilistic automaton, where Q is a finite set of states, qo £ Q is the initial state, A defines the 
transitions and F C Q is the set of accepting states. 

The transitions of a probabilistic automaton are given by a function A : Q x A —> V(Q), 
where A (q,a) is the probability distribution obtained by reading the letter a from the 
state q. The function A induces the function A' : V(Q) x A —» V(Q), where A'(5, a) = 

' A(g, a). Going further, A naturally extends to A* : V(Q) x A* —> T>(Q ) by 
induction: for a letter a £ A, we set A*(d, a) = A'(<5> a), and for an input word u = av, we 
set A *(5,u) = A*(A '(<5,a),v). Intuitively, A *(5,u) is the probability distribution obtained 
by reading the word u starting at the initial probability distribution <5. From now on, we 
will make no difference between A, A' and A*, and denote the three of them by A. 

We denote by FU(s A t ) the probability to go from state s to state t reading u on the 
automaton A, i.e. A (s,u)(t). Then P^(s A T) is defined as At( s A t). Finally, the 

acceptance probability of a word u € A* by A is AA®) ~which we denote by P_ 4 (u). 

For computational purposes, we assume that each value is a rational number given by 
two integers in binary decomposition. 

Definition 1.2 (Value). The value of a probabilistic automaton A, denoted by val(A), is 
the supremum acceptance probability over all input words: 

val(A) = sup PAu). (1-1) 

ueA* 


1.2. The value 1 problem. We are interested in the following decision problem: 

Problem 1.3 (Value 1 Problem). Given a probabilistic automaton A, decide whether 
val(A) = 1. 

The value 1 problem can be reformulated using the notion of isolated cut-point intro¬ 
duced by Rabin in his seminal paper |Rab63| : an automaton has value 1 if and only if the 
cut-point 1 is not isolated. 


a,b a,b 



Figure 1: This automaton has value 1 if and only if x > 
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The automaton depicted on figure Q] lias value 1 if and only if x > \ (a similar example 
appears in [BBG12] ). The input alphabet is A = {a, b}, the initial state is the central state 
0 and the unique final state is L2. 

We describe the behaviour of this automaton. After reading one b, the distribution is 
uniform over L\,R\. To reach L2, one needs to read a b from the state L\, but on the 
right-hand side this leads to the non-accepting absorbing state R2. In order to maximize 
the probability to reach L2, one tries to “tip the scales” to the left. 

If x < i. there is no hope to achieve this: reading a letter a gives more chance to stay 
in R\ than in L\ thus all words are accepted with probability at most and val(A) = 
However, if x > \ then we show that A has value 1. 

We have: 

P^(0 Li) = i ■ x n and P^(0 ^R l ) = y{ 1 - x) n 

We fix an integer N and analyse the action of reading ( ba n ) N ■ b: there are N “rounds”, 
each of them corresponding to reading ba n from 0. In a round, there are three outcomes: 
winning (that is, remaining in L\) with probability p n = \ ■ x n , losing (that is, remaining in 
R2) with probability q n = | • (1 — x) n , or going to the next round (that is, reaching 0 ) with 
probability 1 — (p n + q n ). If a round is won or lost, then the next b leads to an accepting or 
rejecting sink; otherwise it goes on to the next round, for N rounds. Hence: 

p A((ba n ) N -b) = E£Li (l-(p„ + ?n)) fc - 1 -Pn 

n . 1 -( 1 ~(Pn+gn)) Af 

1—(1 -{Pn+qn)) 

= iq4n: • (l - (1 - (Pn + Qn)) N ) 

Pn 

We now set N = 2 n . A simple calculation shows that the sequence ((1 — (pn+qn)) 2 ")n€N 
converges to 0 as n goes to infinity. Furthermore, if x > i then —- < 1, so — = (—' 
converges to 0 as n goes to infinity. It follows that the acceptance probability converges to 
1 as n goes to infinity. Consequently: 

limP A({ba n f n -b) = 1 . 

n 

This example witnesses two surprising phenomena: 

• the value is discontinuous with respect to the transition probabilities, as for x = -7 the 
value is and for x > -7 the value is 1 ; 

• the sequence of words (( 6 a n ) 2 " • b) n witnessing the value 1 involves two convergence 
speeds: indeed, the words a n b are repeated an exponential number of times, namely 2 n . 
One can show that repeating only n times does not lead to words accepted with arbitrarily 
high probability. 


1.3. Recurrent states and idempotent words. We fix A a probabilistic automaton, 
and define two main notions: recurrent states and idempotent words. 

Definition 1.4 (Induced Markov chain). Let u be a finite word, it induces a Markov chain 
Ma, u whose state space is Q and transition matrix M^ u is defined by: 

Ma, u(s,t) = P^t(s — > t). 
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We rely on the classical notion of recurrent states in Markov chains. 

Definition 1.5 (Recurrent state). A state s is u-recurrent if it is recurrent in Mj^ u . 

A finite word u is idempotent if reading once or twice the word u does not change 
qualitatively the transition probabilities. 

Definition 1.6 (Idempotent word). A Markov chain is idempotent if its transition matrix 
M satisfies that for all states s,t: 

M(s,t ) > 0 <=> M 2 (s,t ) > 0. 

A finite word u is idempotent if Ma,u is idempotent. 

In the case of idempotent words, recurrence of a state is easily characterized, relying 
on simple graph-theoretical arguments: 

Lemma 1.7. Let u be an idempotent word. A state s is u-recurrent if and only if for all 
states t we have: 

Ma,u ( s ,*) > 0 => M.A,u(t,s ) > 0 . 

2. An (incomplete) algorithm for THE value 1 PROBLEM 

In this section, we present an algebraic algorithm for the value 1 problem, called the Markov 
monoid algorithm. Since the problem is undecidable, this algorithm does not solve the 
problem on all instances; we will show that it is correct , i.e. if it answers that an automaton 
has value 1, then the automaton does have value 1, but not complete, i.e. the converse does 
not hold. In the next section, we shall show that this algorithm is complete for the class of 
leaktight automata. 

2.1. The Markov monoid algorithm. Our algorithm for the value 1 problem computes 
iteratively a set Q of directed graphs called limit-words. Each limit-word is meant to 
represent the asymptotic effect of a sequence of input words, and some particular limit- 
words can witness that the automaton has value 1. 

Definition 2.1 (Limit-word). A limit-word is a function u : Q 2 —> {0,1}, such that for all 
states s, there exists a state t such that u(s,t) = 1. 

In proofs and examples, we will adopt either of the two equivalent views for limit-words: 
graphs over the set Q or square matrices over Q x Q. 

We now explain the algorithm in detail. For the remainder of this section, wefixda 
probabilistic automaton. Initially, Q only contains those limit-words a that are induced by 
input letters a £ A : 

Vs,t £ Q, (a (s,t) = 1 -<=>■ P.a(s A £) > 0), 
plus the limit-word 1 which is induced by the empty word: 

Vs, t £ Q, (l(s, t) = 1 -<=>• s = t ). 

The algorithm repeatedly adds new limit-words to Q. There are two ways for that: 
concatenating two limit-words or iterating an idempotent limit-word. 
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ALGORITHM 1: The Markov monoid algorithm. 

Data: A probabilistic automaton. 

G t — {a | cl £ A} U {1}. 
repeat 

if there is u, v £ Q such that u • v ^ Q then 
| add u • v to G 

end 

if there is u £ Q such that u is idempotent and ^ Q then 
| add u 8 to G 

end 

until there is nothing to add ; 
if there is a value 1 witness in Q then 
return true; 
else 

return false; 

end 


Concatenation of two limit-words The concatenation of two limit-words u and v 
is the limit-word u • v such that: 

(u • v)(s, t) = 1 3q £ Q , u(s, q) = 1 and v(q, t ) = 1. 

In other words, concatenation corresponds to the multiplication of matrices with coefficients 
in the boolean semiring ({0, 1 },V,A). Intuitively, the concatenation of two limit-words 
corresponds to the concatenation of two sequences ( u n ) nG n and (u„) ng N of input words into 
the sequence (u n ■ v n ) n& ^. 

We say that a limit-word u is idempotent if u u = u. The following lemma gives simple 
properties of idempotent limit-words. 

Lemma 2.2. For all limit-words u: 

• the limit-word ul®l ! is idempotent, 

•if u is idempotent, then for all states r £ Q, there exists a state r' £ Q such that 
u(r, r') = 1 and r' is u-recurrent. 

The proof is omitted and relies on simple graph-theoretical arguments. 

Iteration of an idempotent limit-word The iteration u* *" of a limit-word u is only 
defined when u is idempotent. It relies on the notion of u-recurrent state. 

Definition 2.3 (u-recurrence). Let u be an idempotent limit-word. A state s is u-recurrent 
if for all states t, we have: 

u(s,t) = 1 => u(t,s) = 1 . 

Note that this echoes Lemma [1.71 The iterated limit-word u*- removes from u any edge that 
does not lead to a recurrent state: 

u**(s,t) = 1 -<=> u(s,f) = 1 and t is u-recurrent. 

Intuitively, if a limit-word u represents a sequence (ti n ) ne N then its iteration u^ represents 

the sequence for some increasing function / : N — > N. 

V / n€ N 






N. FIJALKOW, H. GIMBERT, E. KELMENDI, AND Y. OUALHADJ 


2.2. The Markov monoid and value 1 witnesses. The set of limit-words Q computed 
by Algorithm [T] is called the Markov monoid. 

Definition 2.4 (Markov monoid). The Markov monoid associated with A is the smallest 
set of limit-words containing {a | a G A}U{ 1 } and closed under concatenation and iteration. 

Two key properties, consistency and completeness , ensure that the limit-words of the 
Markov monoid reflect exactly every possible asymptotic effect of a sequence of input words. 

Definition 2.5 (Reification). A sequence ( u n ) n of words reifies a limit-word u if for all 
states s,t, (P^(s -^b t)) n eN converges and: 

u(s,t) = 1 -<=> limP_ 4 (s -^b t) > 0. (2-1) 

n 

Note that if (n n ) nS N reifies u, then any subsequence of (u n ) n& ^ also does. We will use 
this simple observation several times. 

Definition 2.6 (Consistency). A set of limit-words Q is consistent with A if for every 
limit-word u € Q, there exists a sequence of input words (ttn)neN which reifies u. 

Definition 2.7 (Completeness). A set of limit-words Q is complete for A if for each sequence 
of input words (u n ) nSN , there exists u£(/ such that for all states s,t € Q: 

limsupP v 4 (s -^b t) = 0 => u(s,t) = 0. (2-2) 

n 

Limit-words are useful to decide the value 1 problem because some of these are witnesses 
that the automaton has value 1. 

Definition 2.8 (Value 1 witness). A value 1 witness is a limit-word u such that for all 
states t: 

u(q 0 ,t) = 1 => t <E F, (2.3) 

where qo is the initial state of the automaton. 

Thanks to value 1 witnesses, the answer to the value 1 problem can be read in a 
consistent and complete set of limit-words: 

Lemma 2.9 (A criterion for value 1). If Q is consistent with A and complete for A, then 
A has value 1 if and only if Q contains a value 1 witness. 

Specifically: 

• If G is consistent with A and contains a value 1 witness, then A has value 1, 

• If Q is complete for A and A has value 1, then A contains a value 1 witness. 

Proof. We prove the first item. Assume that Q is consistent with A and contains a value 
1 witness u. Since Q is consistent, there exists a sequence (tt n )ngN reifying u. ft follows 
from m and (12.31) that for t £ F, we have lim n Pj\(qo —^b t) = 0. Thus lim n P v 4 (n n ) = 
YlteF li m n P.a((7o —~A t) = 1 , so A has value 1 . 

We now prove the second item. Assume that Q is complete for A and that A has 
value 1. Then there exists a sequence of words (u n ) ne n such that lim n P v 4 (n n ) = 1, i.e. 
lim n P^^o “b t) = 1. Since for all n G N, we have P-A^o —^b q) = 1, then 

for all t (f F, limsup n P_ 4 (go — b t) = 0. Since Q is complete, there exists a limit-word u 
such that (|2.2I) holds. Then u is a value 1 witness: let t G Q such that u(qo,t) = 1, then 
according to (12.21) . liirisup n P^(qo —^b t) > 0, hence t G F. □ 
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2.3. Correctness of the Markov monoid algorithm. 

Theorem 2.10. The Markov monoid associated with A is consistent. 


This implies that if the Markov monoid algorithm outputs “true”, then for sure the 
input automaton has value 1. This positive result holds for every automaton (leaktight or 
not). 

To prove Theorem l2.10l recall that the Markov monoid is the smallest set of limit-words 
containing {a | a £ A} U {1} and closed under concatenation and iteration, hence it suffices 
to prove that the initial elements form a consistent set, and the closure under the two 
operations. 

First, a is reified by the constant sequence (a) nS N, and 1 by the constant sequence 
(e)neN- We state the closure under the two operations in the following proposition: 


Proposition 2.11. Let (u n )neN and (u n ) ne N be two sequences that reify the limit-words u 
and v respectively. Then: 

(1) the sequence of words (u n ■ u n ) ne N reifies u • v, 

(2) if u is idempotent, then there exists an increasing function f : N —> N such that for all 


increasing functions g : N —» N satisfying g > f, the sequence [f u 'g( n ' l 
limit-word u : . 


reifies the 

n€ N 


The statement about iteration is stronger than required: the existence of / such that 
('Uj( n )) ne pj reifying the limit-word is enough to prove Theorem 12.101 However, we will 
use this stronger result later on (in Section 15.4p . 

Proof. 

(1) Let w n = u n ■ v n . Then (w n ) n£ ^ reifies u • v, since: 

M t) = J2 F A (s M r) • F A (r M t). 
r£Q 

(2) Consider the Markov chain Ml with state space Q and transition matrix M defined by 
M(s,t) = lim„P^(s M t). Since (u n ) n£ N reifies u, we have u(s, t) = 1 if and only if 
M(s, t ) > 0. First observe that since u is idempotent, the Markov chain A A is aperiodic. 
According to standard results about finite Markov chains, this implies that the sequence 
of matrices (M k )ken has a limit which we denote by M°°, satisfying the following: 

Vs,t 6 Q, M°°(s,t ) > 0 =>- t is recurrent in M. (2.4) 

By definition the sequence of matrices (Af 4 , u „) ngN converges to M. Since the matrix 

product operation is continuous, for every k £ N, ( M k A ) converges to M k . So 

V A “"/neI 

for every k > 1, there exists € N such that for all p > Nk, \\M k — ||oo < p 

We define / : N —> N by induction, so that f(k) is the maximum of f(k — 1) + 1 and 

of Nk, ensuring that / is increasing. Then for any increasing function g : N —> N 

satisfying g > /, the sequence of matrices („ ) converges to M°°. We prove 

V (”)/nGN 
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that U n , s 

9 O) 


reifies u : : 

neN 

u^(s, t) = 1 


^ u(s, t) = 1 and t is u-recurrent 
M(s, t) > 0 and t is recurrent in Ai 
M°°(s,t) > 0 

-<=> limP v 4(s - 3{n) > t ) > 0, 

n 

where the first equivalence is by definition of the iteration, the second holds because 
(iin)nGN reifies u, the third by definition of M°°, and the fourth because 
converges to M°°. 

This concludes the proof. □ 




Note that completeness is not true in general; for instance, one can show that the 
Markov monoid of the automaton represented in figure Q] is not complete. The next section 
gives a sufficient condition for completeness: the leaktight property. 


3. Decidability of the value 1 problem for leaktight automata 
In this section we establish our main result: 


Theorem 3.1. The value 1 problem is decidable for leaktight automata. 

The definition of leaktight automata is given in the next subsection. For now (in this 
section), we are only interested in decidability issues; we will actually prove in Section [4] 
that the value 1 problem is PSPACE-complete for leaktight automata. 

Note that as observed in the literature [BBG1211CSV131 Fijl4| , the value 1 problem for 
probabilistic automata over finite words is equivalent to the emptiness problem for proba¬ 
bilistic Biichi automata with positive semantics, hence we obtain the following corollary: 


Corollary 3.2. The emptiness problem is decidable for probabilistic Biichi leaktight au¬ 
tomata with positive semantics. 


The following theorem proves that the Markov monoid of a leaktight automaton is 
complete; since it is always consistent, by Lemma [2.91 the Markov monoid algorithm solves 
the value 1 problem for leaktight automata. 


Theorem 3.3. If a probabilistic automaton is leaktight then its Markov monoid is complete. 

The remainder of this section is devoted to the proof of Theorem 13.31 We first define 
the leaktight property, and extend the Markov monoid. This extended version allows to 
state an algebraic characterization of the leaktight property. Then, the technical core of the 
proof relies on a subtle algebraic argument based on the existence of ^-factorization trees of 
bounded height [SirriDOL ISim941 IColfM ITorll] . 
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3.1. Leaks. The undecidability of the value 1 problem comes from the necessity to compare 
parallel convergence rates in order to track down vanishing probabilities. Comparing two 
convergence rates may require to compare the decimals of the rates up to an arbitrary pre¬ 
cision, which in turn can encode a Post correspondence problem, hence the undecidability. 

One of the phenomena that makes tracking vanishing probabilities difficult are leaks. 
A leak occurs in an automaton when a sequence of words turns a set of states C C Q into 
a recurrence class C on the long run, but on the short run, some of the probability of the 
recurrence class is “leaking” to a different recurrence class. 



Figure 2: ( a n ■ 6 ) ne pj is a leak from L\ to L 2 . 


Such leaks occur in the automaton depicted in the left hand side of figure [2] with the 
input sequence (a n 6 ) nS N- As n grows large, the probability to reach L 2 from L\ while reading 
the input word a n b vanishes, thus the sets {L\} and {L 2 } are two different recurrence classes 
on the long run [i.e. asymptotically), however on the short run remains a small yet positive 
probability to reach L 2 from L\. 

The right hand side of figure [2] shows the asymptotic behaviour of reading (a n &) ng N- 
Since the automaton in figure []] contains two symmetric parts identical to figure O it 
features one leak on the left hand side and another in the right hand side. As a consequence, 
the real asymptotic behaviour is complex and depends on the compared speeds of these 
leaks. 

An automaton without leak is called a leaktight automaton. In this section we prove 
that the value 1 problem is decidable when restricted to the subclass of leaktight automata. 
The formal definition of a leak is as follows: 

Definition 3.4 (Leak). Let (ri n ) ng N be a sequence of idempotent words. Assume that 
the sequence of matrices P^(w n ) converges to a limit M, that this limit is idempotent and 
denote M the assocaited Markov chain. 

The sequence ( u n ) ne n is a leak if there exist r,q £ Q such that the following three 
conditions hold: 

( 1 ) r and q are recurrent in A4, 

(2) lim n P v 4 (r q) = 0, 

(3) for all n € N, ¥ A (r q) > 0. 

Definition 3.5 (Leaktight automata). A probabilistic automaton is leaktight if it has no 
leak. 

Several examples of leaktight automata are given in Section [5j 
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3.2. The extended Markov monoid. The existence of leaks can be decided by a slight 
extension of the Markov monoid algorithm which keeps track of strictly positive transition 
probabilities. 

Definition 3.6 (Extended limit-word). An extended limit-word is a couple (u, u+) of two 
limit-words, such that for all s,t € Q, we have u(s,i) = 1 => u+(s,t) = 1 . 

As for limit-words, extended limit-words can be seen either as graphs over the set Q, 
or couples of square matrices over Q x Q. Such a graph has two different kind of edges: an 
edge (s,t) is “normal” if u (s,t) = 1 , and is a +-edge if u (s,t) = 0 but u + (s,t) = 1 . 

We define the concatenation and iteration operations for extended limit-words. The 
concatenation of two extended limit-words (u, u + ) and (v, v+) is the component-wise con¬ 
catenation, i.e. (u • v, u + • v + ). The iteration of an extended limit-word (u, u+) is only 
defined when it is idempotent (i.e. component-wise idempotent), by (u, u + )^ = (u#,u+). 

Definition 3.7 (Extended Markov monoid). The extended Markov monoid is the small¬ 
est set of extended limit-words containing {(a, a) | a £ A} U {(1,1)} and closed under 
concatenation and iteration. 

Note that if (u, u+) is in the extended Markov monoid, then u is in the Markov monoid. 
The essential difference between the Markov monoid and its extended version is that 
the extension keeps track of those edges that are deleted by successive iteration operations. 
This serves two purposes: first, to characterize the leaktight property in algebraic terms, 
and second, to prove Theorem 13.31 

We state a consistency result for the extended Markov monoid, extending Theorem l2.ini 
The proofs of both these results are similar and given only once. 

Lemma 3.8. For each (u, u+) in the extended Markov monoid, there exists a sequence 
(rt n ) nS N such that for all states s,t £ Q, (P^(s f)) n eN converges and: 

u (s,t) = 1 <(=> limPMs t) > 0, (3-1) 

n 

for all n € N, ^u + (s,t) = 1 <(=^ P^(s t) > o'j . (3-2) 


3.3. Leak witnesses. 

Definition 3.9 (Leak witness). An idempotent extended limit-word (u, u+) is a leak witness 
if there exist r,q £ Q such that the following three conditions hold: 

(1) r and q are u-recurrent, 

(2) u (r,q) = 0, 

(3) u+(r,q) = 1. 

Lemma 3.10. If a probabilistic automaton is leaktight, then its extended Markov monoid 
does not contain any leak witness. 

Proof. Suppose that there is a leak witness (u, u+) in the extended Markov monoid: u and 
u_|_ are idempotent and there exists r,q £ Q such that r and q are u-recurrent, u(r, q) = 0 
and u + (r, q) = 1. We prove that there exists a leak. 

Thanks to Lemma [3781 there exists a sequence (u n ) nS M satisfying (13.111 and (13.21) . Note 
that since u_|_ is idempotent, (I3.2|) implies that for all n € N, u n is idempotent. 
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Consider the Markov chain A4 with state space Q and transition matrix M defined by 
M(s,t ) = lim n P_ 4 (s t). A4 is idempotent since u is idempotent and thanks to (13.111 . 

We show that (u n ) ne ^ is a leak. There are three conditions to be met. 

First, r and q are recurrent in M: this follows from m and the fact that r and q 
are u-recurrent. Second, lim n P_ 4 (r -^b q) = 0: this follows from (13.111 and the fact that 
u (r, q) = 0. Third, for all n € N, P^i(r -^b q) > 0: this follows from (13.2jl and the fact that 
u+(r,g) = l. □ 

As we will show in the next section, the converse of Lemma 13.101 is also true, which 
gives an algebraic characterization of the leaktight property using the extended Markov 
monoid. However, the proof of the converse implication is more involved and requires the 
lower bound lemma (Lemma 13. 15p . which is the object of the next subsection. 

3.4. Stabilization monoids and ((-factorization trees. We now introduce the technical 
material required to state and prove the lower bound lemma. The key notions here are 
stabilization monoids and ((-factorization trees. 

Factorization trees for monoids have been introduced by Simon |Sim90j . Roughly speak¬ 
ing, Simon’s factorization theorem states that given a morphism <ft : A* —>• M from the set 
of finite words over A to a finite monoid M, the following holds: for all words u, the com¬ 
putation of cj)(u) can be factorized in a tree whose depth is bounded independently of the 
length of the word. 

Simon later developed the notion of decomposition trees to solve the limitedness prob¬ 
lem for distance automata |Sim94j . To this end, he defined an iteration operation (J for 
monoids over the tropical semiring (N U {oc}, min, +). Then Kirsten extended this tech¬ 
nique to desert automata and the nested distance desert automata IKirOoi . After him, 
Colcombet generalized this approach by defining stabilization monoids |Col09j . which are 
monoids equipped with an iteration operation, and proved the existence of ((-factorization 
trees of bounded depth. The formal definition is as follows: 

Definition 3.11 (Stabilization monoid). A stabilization monoid (M, •, (j) is a finite monoid 
(M, •) equipped with an iteration operation jj : E(M) —> E(M), where E(M) is the set of 
idempotents of M, such that: 

(a ■ b) 1 ^ ■ a = a ■ (b ■ a,y for a ■ b E E(M) and b ■ a £ E(M ), (3-3) 

( e B)tt _ e tt f or e g e(M ), ( 3 - 4 ) 

e tt . e _ gtt for e G E(M). (3.5) 

Lemma 3.12. The extended Markov monoid is a stabilization monoid. 

Proof. To start with, the extended Markov monoid is a monoid for the concatenation: 1 is 
the neutral element, and the concatenation is associative. 

Now, let us prove the three properties required for the iteration operation (j. 

Proof of (13.311 . Let (u, u + ), (v, v+) such that (u • v,u + • v+) and (v • u, v + • u+) are 
idempotent. By definition ((u, u + ) • (v, v+))^ • (u, u + ) is equal to ((u • v)^ • u, u + • v + • u+), 
and (u, u+) • ((v, v + ) • (u, u+))^ to (u • (v • u)N, u + • v + • u + ) . Let s,t <E Q, we have the 
following equivalence: ((u • v)^ • u) ( s,t ) = 1 if and only if: 

there exists r,q £ Q, u(s, r) = 1 A v(r, q) = 1 A u (q, t) = 1 A q is (u • v)-recurrent, (3.6) 
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and similarly, (u • (v • u)^) (s, t) = 1 if and only if: 

there exists r,q G Q, u(s, r) = 1 A v(r, q) = 1 A u(q, t) = 1 A t is (v • u)-recurrent. (3.7) 

We show that (13.61) and (13.71) are equivalent. Assume (13.61) . and prove that t is (v • u)- 
recurrent. Let p £ Q such that (v • u)(t,p) = 1. Since v is a limit-word, there exists 
£ G Q such that v(p,£) = 1 . Observe that u (q,t) = 1 , (v • u)(t,p) = 1 and v(p,£) = 1, so 
(u • v) 2 (g,£) = 1. As u • v is idempotent, this implies (u • v)(q,£) = 1. Since q is (u • v)- 
recurrent, we have (u • v)(£, q ) = 1. Altogether, v(p, l) = 1, (u • v)(£, q) = 1 and u(g, t) = 1 
imply that (v • u) 2 (p, t) = 1. As v • u is idempotent, this implies (v • u )(p,t) = 1, so t is 
(v • u)-recurrent, and (13.71) is proved. Conversely, assume (13.71) . and prove that q is (u • v)- 
recurrent. Note that from v(r, q) = 1, u(g, t) = 1 and the fact that t is (v • u)-recurrent, we 
obtain that (v • u )(t,r) = 1. Let p G Q such that (u • v)(q,p) = 1. Since u is a limit-word, 
there exists £ G Q such that u (p,£) = 1. Observe that v(r,q) = 1, (u • v)(q,p) = 1 and 
u (p,£) = 1 , so (v • u) 2 (r, £) = 1 , and with (v • u)(t,r) = 1 this implies (v • u) 3 (f,T) = 1 . 
As v • u is idempotent, this implies (v ■ u)(t,£) = 1. Since t is (v • u)-recurrent, we have 
(v • u)(£,t) = 1. Altogether, u (p,£) = 1, (v • u)(£,t) = 1, (v • u)(t,r) = 1 and v{r,q) = 1 
imply that (u • v) 3 (p, q) = 1. As u • v is idempotent, this implies (u • v)(p, q) = 1, so q is 
(u • v)-recurrent, and (13.61) is proved. The property (13.31) follows. 

Proof of ()3.4p . This boils down to proving (u^) : = iA This is clear from the definition 
of A, since the notions of u-recurrence and lArecurrence coincide. 

Proof of (|3.5j) . This boils down to proving A • u = A. It follows from the observation 
that if r £ Q is u-recurrent and u(r, t) = 1, then t is u-recurrent (under the assumption 
that u is idempotent). □ 

Definition 3.13. Let A be a finite alphabet, (M, -, fl) a stabilization monoid and (j) : A* —> 
M a morphism into the submonoid (M, •). A jj- factorization tree of a word u S A* is a finite 
unranked ordered tree, whose nodes have labels in A* x M and such that: 

i) the root is labelled by (u, u), for some u € M, 

ii) every internal node with two children (called concatenation nodes) labelled by (rti, ui) 
and (u 2 , U 2 ) is labelled by (u± ■ U 2 , ui • U 2 ), 

iii) every internal node with three or more children (called iteration nodes) is labelled by 
(ui ... u n , e®) for some e € E(M ), and its children are labelled by (u\, e),..., (u n , e). 

iv) every leaf is labelled by (a, a) where a is a letter, or (e, 1 ). 

Note that in a factorization tree, the second label is not always the image of the first 
component under <f>\ indeed, it is an element of the stabilization monoid (M, •,(j) whereas 
the image of a finite word under <f> is an element of the submonoid (M, •). However, the 
projection of second label into this submonoid (which consists in ignoring the operation (j) 
is indeed the image of the first component under (f>. 

The following theorem was stated for the tropical semiring in [Sim94j . and generalized 
in [Col09] . A simple proof can be found in [Torllj . 

Theorem 3.14. Let A be a finite alphabet, (M, •, ft) a stabilization monoid and (j) : A* —>• M 
a morphism into the submonoid (M, •). Every word u € A* has a [j -factorization tree whose 
depth is less than 3 ■ \M\. 
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3.5. The lower bound lemma. We are ready to state and prove the lower bound lemma, 
which is the central argument in the proof of completeness of leaktight Markov monoids. 

Lemma 3.15 (Lower bound lemma). Let A be a probabilistic automaton whose extended 
Markov monoid contains no leak witness. Let ! p m \ n the smallest non-zero transition proba¬ 
bility of A. Then for all words u € A*, there exists (u, u + ) in the extended Markov monoid 
such that, for all states s,t: 


U+(s, t) = 14 = 

=y- P^(s —> t) > 0, 

(3.8) 

u(s, t) = 1 = 

, u x oslOI ^ 1 

=4 P a(s~H)> pb D 

(3.9) 


Proof. Consider a finite word u G A*; by Theorem 13.141 applied to the extended Markov 
monoid Q+ associated with A (which is a stabilization monoid thanks to Lemma 13.121) and 
the morphism f> : A —>• M defined by 4>(a) = (a,a), there exists a ^-factorization tree of 
depth at most 3 • \G+\, whose root is labelled by (it, (u, u + )) for some extended limit-word 
(u,u+). 

The depth of a node in this tree is defined in a bottonr-up fashion: the leaves have 
depth zero, and a node has depth one plus the maximum of the depths of its children. 

We prove by a bottom-up induction (on h) that for every node (it, (u, u + )) of this tree 
at depth h, for all states s,t: 

u_| -(s,t) = 1 -<=> IPU( S —> t) > 0, (3.10) 

u (s,t) = 1 Pa(« Af)> in- (3.11) 

The case h = 0 is for leaves. Here, either a is a letter a and u = u + = a, or u is the 
empty word e and u = u + = 1. Then both (13.101) and (13.111) hold. 

Assume h > 0, there are two cases. 

First case: a concatenation node labelled by (a, (u, u+)) with two children labelled 
by (u±, (ui, u +i i)) and (it 2 , (U 2 , u +i 2 )). By definition u = u\ ■ U 2 , u = ui • U 2 and u + = 
u +j i • u +;2 . 

We first prove that (|3.10l) holds. Indeed, for s,t £ Q, u +(s,t) = 1 if and only if there 
exists r £ Q such that u +j i(s, r) = 1 and u +!2 (r, t) = 1. On the other side, since: 

P_ 4 ( s 4 t) = J_] P^( S r) • P^(r ^b t), 
r£Q 

then P^(s -A i) > 0 if and only if there exists r <E Q such that P^i(s —4 r) • P^(r —A t) > 0, 
which is equivalent to P^(s —4 r) > 0 and P_^(r ~b t) > 0. We conclude with the induction 
hypothesis. 

Now we prove that (13.111) holds. Let s,t £ Q such that u(s,£) = 1. Then there exists 
r £ Q such that ui(s,r) = 1 and u 2 (r, t) = 1. So: 

Pm(s b t) > P^(s ^b r) • P^(r ^b t) > p^ in ■ p^ n = , 

where the second inequality is by induction hypothesis. This completes the proof of (13.111) . 

Second case: an iteration node labelled by (it, (u“,u_|_)) with k sons labelled by 
(iti, (u, u + )),..., (it*., (u, u + )). By definition, u = u\ ■ ■ ■ Uk, and (u, u+) is idempotent. 

The proof that (13.101) holds is similar to the concatenation node case. 
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Now we prove that (13.111) holds. Let s,t £ Q such that tr(s,f) = 1. Since k > 3: 

P^(s bt)> P^(s ^b t) • ^ F A (t Uk ~ - > q ) • P^(g -^4 f). (3.12) 

q eQ 

To establish (13.111) we prove that: 

Pa(s t) > ptin, (3.13) 

for all qeQ, ¥ A (t - 2 Uk ~ - > q) > 0 => P A {q ^b t) > in . (3.14) 

We prove (13.131) . Since u^(s,i) = 1, by definition u (s,t) = 1 and t is u-recurrent. The 
induction hypothesis for the node (tq, (u, u+)) implies that P.a(s -4- 1) > p^ in , i.e. (13.131) . 
Now we prove ()3.14l) . For that we use the hypothesis that (u, u+) is not a leak witness. 

Let q £ Q such that ¥ A (t —-——A q) > 0. By induction hypothesis for each child, (13.101) 
implies that u^T 2 (t, q) = 1. Since u + is idempotent, u+(f, q) = 1. We argue that u(g, t) = 1. 
Let t £ Q a u-recurrent state such that u (q,£) = 1. Then u+(i, ■£) = 1, and t,i are u- 
recurrent. Since (u, u + ) is not a leak witness, it follows that u (t,£) = 1, which implies that 
u (£,t) = 1 since t is u-recurrent. Together with u (q,£) = 1, this implies u (q,t) = 1. Thus, 
by induction hypothesis and according to (13.111) . P A (q ^b t ) > p^ in , so (13.141) holds. 

Now, putting (13.121) . (13.131) and ()3.14[) altogether: 

p a(s bt)> P^(s ^b t) ■ y~] F A (t - 2 Uk ~-> q) ■ F A (q -b t ) 

QG Q 

nh tit , /, U 2'" U k-l \ Oh 

> Pmin ■2^ P b t -> q) ■ Pmin 

q&Q 

= V 2h+1 
Tmin i 

where the last equality holds because Yl q eQ F A (t — —-—4- q) = 1. This completes the proof 
of (ETTTll . 

To conclude, note that Q+ has less than 3^ elements. □ 


3.6. Completeness of the Markov monoid algorithm for leaktight automata. In 

this subsection we rely on the lower bound lemma (Lemma 13.151) to prove Theorem 13.31 
Let A be a leaktight automaton. By Lemma 13.101 its extended Markov monoid does not 
contain any leak witness, hence Lemma 13.151 applies. 

We prove the completeness of the Markov monoid associated with A. Let (it n )neN be 
a sequence of finite words. By Lemma 13.151 for each word u n there exists (u n ,u+ in ) in the 
extended Markov monoid such that for all states s, t: 

it Q 3lQI^+l 

U n (s,t) = 1 = 4 > P^(S -bt)> p 2 m in 

Since the set of limit-words is finite, there exists N £ N such that {n £ N | ujv = u n } is 
infinite. To complete the proof, we prove that ujv satisfies, for all states s,t: 
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Un Q3IQI +1 

Assume lim sup P.4 (s — A t) = 0, then limsupP^s — A t) < p ^ n for n sufficiently large. 
Since u n = u n for infinitely many n £ N, this implies uj\r(s,t) = 0, which completes the 
proof of Theorem 13.31 


4. Properties of leaktight automata 

In this section, we extend the algorithm presented in Section [2l and investigate its running 
complexity. The extended algorithm has two features: first, it checks at the same time 
whether an automaton is leaktight and whether it contains a value 1 witness, second, it 
runs in polynomial space. 

We present an algebraic characterization of the leaktight property based on the extended 
Markov monoid, allowing the extended algorithm to check the leaktight property. For the 
complexity, one needs a deeper understanding of the Markov monoid; in this section, we will 
show a linear bound on the JJ-height, allowing to compute the extended Markov monoid in 
polynomial space. As a corollary, we obtain that the value 1 problem for leaktight automata 
is PSPACE-complete. 

4.1. Characterization of the leaktight property. In this subsection, we show the con¬ 
verse of Lemma 13.101 which implies the following theorem, characterizing the leaktight 
property in algebraic terms. 

Theorem 4.1. An automaton A is leaktight if and only if its extended Markov monoid does 
not contain any leak witness. 

Lemma 13.151 is instrumental in the proof of this lemma. 

Proof. We prove that if the extended Markov monoid of an automaton A does not contain 
any leak witness, then A is leaktight. The converse was proved in Lemma 13.101 

Assume A has a leak (u n ) nS N) we show that its extended Markov monoid contains a 
leak witness. Consider the Markov chain M with state space Q and transition matrix M 
defined by M(s,t ) = lim n P_4(s ^A t). By assumption M is idempotent. 

By definition of a leak: 

r and q are recurrent in A4, (4-1) 

M(r,q) = 0, (4.2) 

for all n € N, P A {r ^A q) > 0. (4.3) 

Assume towards contradiction that the extended Markov monoid does not contain any 
leak witness, then Lemma 13.151 applies. For each word u n , there exists (u n ,u +)n ) in the 
extended Markov monoid such that for all states s, t: 

u + >n (s,t) = 1 lPU(s ^At)> 0, (4.4) 

,|Q| 2 + 1 

u n (s,t) = 1 =k Pa(s t) > P 2 m in • (4.5) 

Since the extended Markov monoid is finite, there exists N £ N such that for infinitely 
many n £ N, we have (ujy, u +i jy) = (u n , u +>n ). 

Note that since each u n is idempotent, (14.4[) implies that each u +;n is idempotent as 

well. 
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Let (v, v + ) = (u/v, The power Q|! ensures that u^’ 

Lemma 12.21 Since u +i tv is idempotent, v + = u +! tv- Also, since v is 
exists r' and q' which are v-recurrent, such that v(r, r') = 1 and v(q, q') 
to Lemma 12.21 

Now, we prove that (v, v + ) is a leak witness: 

is idempotent, by 
idempotent, there 
= 1 , again thanks 

r' and q' are v-recurrent, 

(4.6) 

II 

o 

(4.7) 

V+(/,?') = 1. 

(4.8) 


o 3 IQI 2 +i , , 

Let V = Pmin and K = |Q|!. 

Observe that for all states s,t, we have v(s,t) = 1 =>• M(s,t) > 0: 
v(s,t) = 1 


u %(s,t) = 1 

u n(s,t) = 1 for infinitely many n 
u k 

Pyl(s — t) > r/ x for infinitely many n 
limP v 4 (s -^4 t) > T] h 

n 

M K (s,t ) > 0 
M(s, t) > 0 


(by definition of v) 
(by definition of N) 

(by dH5])) 


(by definition of M ) 
(since M is idempotent). 


First, (|4.6D is by definition of r' and q'. 

We prove (14.71) . Towards contradiction, assume that v(r',q') = 1. Then M(r',q ') > 0, 
so together with M(r,r') > 0 (which follows from v(r, r') = 1) this implies M 2 (r, q') > 0, 
so M(r,q') > 0 as M is idempotent. Since M(q,q r ) > 0 (which follows from v(q,q') = 1) 
and q is recurrent in M, we have M(q',q ) > 0. This implies M 2 (r,q) > 0, and M(r,q) > 0 
because M is idempotent, which contradicts (14.21) . 

We prove (14.81) . Thanks to (14.3p and (|4.4j) . we have u +! tv(f, q) = 1, i.e. v+(r,q) = 
1. Since M(r,r') > 0 and r is recurrent in M, we have M(r',r) > 0, so (14.41) implies 
that u +! tv(f / , r) = 1, i.e. v + (r',r) = 1. Similarly, M(q,q') > 0, so (14.41) implies that 
u + i at((/, (/) = 1, i.e. v + (q,q') = 1. The three equalities v + (r',r) = 1, v + (r, q) = 1 and 
v_(-(<?,</) = 1 imply v\(r',q') = 1 , and since v + is idempotent v + (r',g') = 1 . 

It follows that (v, v + ) is a leak witness, which completes the proof. □ 


The immediate corollary of Theorem 14.11 is that checking whether an automaton is 
leaktight can be done by computing the extended Markov monoid and looking for leak 
witnesses, hence it is decidable. 


4.2. The extended Markov monoid algorithm. Algorithm [2] computes the extended 
Markov monoid, and looks for value 1 witnesses, which in the extended Markov monoid is 
an extended limit-word (u, u + ) such that u is a value 1 witness (in the Markov monoid). If 
there is a value 1 witness, then the automaton has value 1 , even if it is not leaktight, thanks 
to Theorem 12.101 Otherwise, the algorithm looks for a leak witness; if there is no leak 
witness, then the automaton is leaktight thanks to Theorem 14.11 and it does not have value 
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1 thanks to Theorem 13.31 In case there is a leak witness, the automaton is not leaktight, 
and nothing can be said. 


ALGORITHM 2: The extended Markov monoid algorithm. 

Data: A probabilistic automaton. 

G+ t— {(a, a) | a £ A} U {(1,1)}. 
repeat 

if there is (u, u+), (v, v+) £ Q+ such that (u • v, u + • v+) ^ Q+ then 
add (u v, u + • v+) to Q + 

end 

if there is (u, u+) £ Q+ such that (u, u+) is idempotent and (u**,u+) ^ £?+ then 
add (u 11 , u+) to Q + 

end 

until there is nothing to add ; 
if there is a value 1 witness in Q+ then 
return true; 
else 

if there is no leak witness in Q+ then 
return false; 
else 

return fail: the automaton is not leaktight; 

end 

end 


4.3. Parallel composition and PSPACE-hardness. The objective of this subsection is 
to prove the PSPACE-hardness of the value 1 problem for leaktight automata. To this end, 
we give a reduction from the emptiness problem of n deterministic automata. To prove that 
the reduction indeed constructs leaktight automata, we need to show that deterministic 
automata are leaktight, and the closure under parallel composition. 

Proposition 4.2. Deterministic automata are leaktight. 

Proof. For all limit-words u £ {a | a £ A} U { 1 }, for all states s, there exists a unique 
state t such that u(s, t) = 1. In particular, each recurrence class is formed of only one state 
with a self-loop. This property is preserved by concatenation, and implies that the iteration 
operation is trivial, i.e. = u. Consequently, for all extended limit-words (u, u+) in the 
extended Markov monoid, we have u = u + , which implies that there are no leak witnesses. 

□ 

Definition 4.3 (Parallel composition). Consider two probabilistic automata, denoted A = 
(Q a , q^-, A a , F a ) and B = (Q B ,q B , A B , F B ). We assume that Q A and Q B are disjoint. 

The parallel composition of A and B is: 

A || B = (Q a td Q b , <5 0 , A , F a U F b ), 
where <5o = | ■ q A + 5 • q B , and: 


A(q,a) 


&A{q,a) 

A B (q,a) 


if q e Qa, 
if q e Qb- 
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By definition, for u G A *, we have P^||e(it) = \ • P^(rt) + ^ • P b{u). Note that in 
this definition, we allowed an initial probability distribution rather than only one initial 
state. This could be avoided by adding a new initial state that leads to each previous initial 
state with probability half, but we do it here for technical convenience in the proof of the 
following proposition. 

Proposition 4.4. The leaktight property is stable by parallel composition. 

Proof. The extended Markov monoid 3+^ of the parallel composition embeds into the 
direct product G+ x G+ of the extended Markov monoids of each automaton. 

Note that for (u, u+) G G+^ B , if u(s,f) = 1, then either s, t G Q^ or s,t G and 

similarly for u + . Relying on this, we map ( 11 , 114 . ) G G+ lB to ((u,u+)[M] , (u,u+)[B]), 
where (u,u + )[M] is the restriction to A and similarly for B. An easy induction on (u, u + ) 
shows that this map is an embedding into G+ x G+- 

Consequently, the extended Markov monoid of the parallel composition contains a leak 
witness if and only if one of the extended Markov monoid contains a leak witness. 

Now that we proved that deterministic automata are leaktight, and the closure under 
parallel composition, the PSPACE-hardness of the value 1 problem for leaktight automata 
is easy. 

Proposition 4.5. The value 1 problem for leaktight automaton is PSPACE-/iard. 

Proof. We give a reduction from the following problem: given n deterministic automata 
over finite words, decide whether the intersection of the languages they accept is empty. 
This problem is PSPACE-hard [Koz77] . 

The reduction is as follows: given n deterministic automata, we construct the parallel 
composition of the n automata, where each copy is reached with probability This au¬ 
tomaton has value 1 if and only if the intersection of the languages is not empty, and is 
leaktight by Proposition 14.21 and Proposition 14.41 □ 

4.4. Bounding the ((-height in the Markov monoid. We now consider the running 
complexity of the extended Markov monoid algorithm. A naive argument shows that it 
terminates in less than 3^1 iterations, since each iteration adds a new extended limit-word 
in the monoid and there are at most 3^ different limit-words. This gives an EXPTIME 
upper bound. 

A better complexity can be achieved by looking for a value 1 witness or a leak witness 
in a non-deterministic way. The algorithm guesses the witness by its decomposition into 
concatenations and iterations. The key observation, made by Kirsten |Kir05] in the context 
of distance desert automata, is that the ((-height, that is the number of nested applications 
of the iteration operation, can be restricted to at most |Q|. 

Note that when dealing with ((-height, it suffices to consider limit-words instead of 
extended limit-words, as by definition the second component of an extended limit-word 
does not contain any ((. 

Formally, we define the ((-hierarchy inside the Markov monoid as follows: 

So = ({ a | 0 , G A} U {1}), 

S p+1 = (S p U{J I u G E(S P )}), 
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where (T) is the set of limit-words obtained as concatenation of limit-words in T. 

Definition 4.6 (((-height of a limit-word). The ((-height of a limit-word u is the minimal p 
such that u G S p . 

Theorem 4.7. Every limit-word, has (j -height at most \Q\, i.e. the (J -hierarchy collapses at 
level \Q\. 

In the following, we adapt Kirsten’s proof from |Kir05j to the setting of probabilistic 
automata. Roughly speaking, the proof consists in associating a quantity to each idempotent 
element of the Markov monoid, and to show the following: 

• the quantity is bounded above by |Q|. 

• the quantity strictly decreases when iterating an unstable limit-word (i.e. if ^ u), 

• the quantity does not increase when concatenating. 

Let u be an idempotent limit-word, we define ~ u the relation on Q by s ~ u t if 
u(s,t) = 1 and u(t,s) = 1. Clearly, ~ u is symmetric, and since u is idempotent, ~ u is 
transitive. If for some state s there exists a state t such that s ~ u t, then s ~ u s since u is 
idempotent. Consequently, the restriction of ~ u to the set 

Z u = {s £ Q \ s ~ u s} 

is reflexive, i.e. ~ u is an equivalence relation on Z u . From now on by equivalence class 
of ~ u we mean an equivalence class of ~ u on Z u . We denote by [s] u the equivalence class 
of s, and by Cl(u) the set of equivalence classes of ~ u . The quantity associated with u is 
|Cl(u)|, the number of equivalence classes of ~ u , that is the number of non-trivial connected 
components in the underlying graph of u. Note that |Cl(u)| < IQI- 
Here are two useful observations. 

Lemma 4.8. 

• Let u, v be two limit-words and s,t,r £ Q. Then (u • v)(s,t) > u (s,r) ■ v(r,t). 

• Let u be an idempotent limit-word and s,t € Q. There exists r £ Q such that u (s,t) = 
u(s, r ) • u(r, r) • u(r, t). 

Proof. The first claim is clear and follows from the equality: 

(u -v)(s,t) = 5^u(s,r) • v(r,t). 

r&Q 

Consider now the second claim. For all states r G Q, since u is idempotent we have: 

u(s,t) = u 3 (s,f) = U ( S )P) • u (P,q) ' > u(s,r) • u(r,r) • u(r,t). 

p,q£Q 

Since u is idempotent, we have u = u n+2 , so there exist s = ro,... ,r n +2 = t such that 
u(s,t) = u(ro, n) • • • u(r n+ i, r n+ 2 )- By a counting argument, there exist i,j such that 
1 < * < j < (n + 1) and r* = rj, denote it by r. We have: 

u(s,r) = u*(s, r ) > u(r 0 ,ri) • ■ ■ u(r;_i, rj), 

u(r, r) = u j ~ l (r, r) > u (n, r i+1 ) ■ ■ ■ u(rj_i ,rj), 

u (r,t) = u n+2 ~ j (r,t) > u(rj,r j+1 ) ■ ■ ■ u(r n+ i,r n+2 ). 

Hence, u(s,r) • u(r, r) ■ u(r, t) > u(ro,ri) • • • u(r n+ i, r n+ 2 ) = u(s,t), and the second claim 
follows. □ 
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The following lemma shows that the quantity |Cl(u)| strictly decreases when iterating 
an unstable limit-word (i.e. if 7 ^ u). 

Lemma 4.9. Let u be an idempotent limit-word. 

Cl(u # ) C Cl(u), (4.9) 

if u / u*-, then Cl(u**) / Cl(u). (4.10) 

Proof. We prove (14.91) . Let s £ Z u #; by definition we have s ~ u # s. We show that [s] u # = [s] u . 
For all states t £ Q such that s ~ u # t, we have s ~ u t, so [s] u # C [s] u . Conversely, let t £ [s] u ; 
we have u(s,f) = 1 and u(t, s) = 1. Since s ~ u # s, we have u^(s, s) = 1. So u # M) = 
(u# -u)(s, t) > u#(s, s)-u(s, t ) = 1 , and similarly u#(i, s) = (u-u^)(t, s ) > u(t, s) -u#(s, s) = 1 . 
Thus s ~ u # t, i.e. t £ [s] u #, which concludes to the equality [s] u # = [s] u . In other words, 
the equivalence classes for are also equivalence classes for u, so Cl(u#) C Cl(u). 

We prove (14.101) . Assume u / u^; let s,t such that u(s,i) = 1 and u$(s,t) = 0. By 
Lemma [4.81 there exists r such that u(s,t) = u(s, r) -u(r, r) • u(r, t), so u(r, r) = 1. Towards 
contradiction, assume u^(r, r) = 1. It follows that: 

u^(s, t) = (u • • u)(s, t) > u(s, r) ■ u**(r, r) • u(r, t) = u(s, r) • u(r, r) ■ u(r, t) = 1, 

i.e., v$(s,t) = 1 which is a contradiction. Consequently, u^(r, r) = 0, so r ~ u r and r 9 ^# r. 
Thus, r € Z u but r ^ Z u $. Hence, there is a class [r] u in Cl(u), but there is no class [r] u # 

in Cl(u»). □ 

The following lemma shows that the quantity |Cl(u)| is common to all idempotents in 
the same jT-class. The notion of ^7-class is a classical notion for the theory of monoids, 
derived from one of the four Green’s relations called the jGpreorder (for details about 
Green’s relations, see [Lal79l fH ow95] . or [Colli ] for its applications to automata theory). 

Define u <j v if there exist a, b such that a • v • b = u, and u^v, i.e. u and v are in 
the same , 7 -class, if u <j v and v <j u. 

Lemma 4.10. Let u, v be idempotent limit-words. If u v, then |Cl(u)| < |Cl(v)|. 

Proof. Let a, b two limit-words such that a • v • b = u. First, without loss of generality we 
assume that a • v = a and v • b = b. Indeed, if a and b do not satisfy these conditions, then 
we consider a = a • v and b = v • b. 

We construct a partial surjective mapping /? : Cl(v) -A Cl(u), which depends on the 
choice of a and b. For all states s £ Z v and t £ Z u satisfying a(f, s) ■ v(s, s) ■ b(s, t) = 1 we 
set /3([s] v ) = [t] u • To complete the proof, we have to show that (5 is well defined and that 
(3 is indeed surjective. 

We show that (3 is well defined. Let s, s' £ Z v and t, t' £ Z u , and assume a (t, s ) • v(s, s) ■ 
b (s,t) = 1 and &(t', s')-v(s', s')-h(s r , t') = 1. By definition/3([s] v ) = [t] u and/?([s'] v ) = [f'] u . 
To show that (3 is well defined, we have to show that if [s] v = [s'] v , then [t] u = [f'] u . 
Assume [s] v = [s'] v , i.e., s s', so v(s,s') = 1. Since a (t,s) ■ v(s,s) ■ b (s,t) = 1, we have 
a (t,s) = b (s,t) = 1. Similarly, a (t',s') ■ v(s',s') ■ b(s',t') = 1, so a (t',s') = h(s',t') = 1. 
Consequently, a(t, s) ■ v(s, s') ■ b (s', t') = 1, so u (ft, t') = (a ■ v • b ){t, t') = 1. Symmetrically, 
a (t',s') ■ v(s',s) ■ b(s,t) = 1 , so u(t',t) = 1 , concluding to t ~ u t', i.e. [t] u = [t'] u . 

We show that (3 is surjective. Let t £ Z u . We exhibit some s such that /3([s] v ) = [t] u - 
Since u = a • v • b, there are p,q such that a (t,p) ■ v(p,q) • b (q,t) = u (t,t) = 1, so 
a (t,p) = v(p,q) = b (q,t) = 1. By Lemma f4.8l there exists s such that v(p, s) ■ v(s,s) ■ 
v(s,q) = v(p,q) = 1, so, v(p, s) = v(s,s) = v(s,q) = 1. We have a (t,s) = (a -v)(t,s) > 
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a (t,p) ■ v(p,s) = 1, and b (s,t) = (v • b)(s,f) > v(s,q ) • h(q,t) = 1. To sum up, a (t,s) ■ 
v(s,s) -b(s,t) = 1 , and hence, /3([s] v ) = [t] u- □ 

The following lemma wraps up the previous two lemma. For technical convenience, we 
set S- 1 = 0. 

Lemma 4.11. Let u be an idempotent limit-word and p > 0. If u £ S p \ S p - 1 , then 
|Cl(u)| < \Q\ — p. 

Proof. We proceed by induction on p. For p = 0, the assertion is obvious. Let p > 0, we 
show the claim for p+l. Let u be an idempotent limit-word such that u £ Sp+i \ S p . By 
definition, u = Vi • • • v*. where for all i, either Vj £ S p or v* = u' for u* £ S p and v* ^ S p . 

If for all i, Vj £ S p , then u = vi ■ • • v& £ S p , which is a contradiction. Consequently, 
there exists i such that v* = u' for u.j £ S p and v,; ^ S p . Since Uj € S p and Vj = u? ^ S p , 
we have uj 7 ^ uj. Towards contradiction, assume u* £ S p - 1 , then p > 1 , and this implies 
u- £ S p , which is a contradiction. Hence, Uj £ S p \ S p -\. 

By induction, we have |Cl(uj)| < \Q\ — p. Since u- / u*, by Lemma 14.91 we have 
|Cl(u-)| < |Cl(uj)|. Since u <j u|, bv Lemma l4.1Ul we have |Cl(u)| < |Cl(u*j)|. Altogether, 
it follows |Cl(u)| < \Q\ — (p + 1). □ 

It follows from Lemma H.lll that 5 |q| = £|q| +1 , i.e. the jj-hierarchy collapses at level 
|<2|, proving Theorem 14.71 

The bound is almost tight, as shown in figure [3j The only value 1 witness of this 
automaton is (•••((aQ ai)N a. 2 )^ a 3 )^ ■■■a n _i)^, whose ((-height is \Q\ — 2. Note that this 


1 5 2 



Figure 3: A leaktight automaton with value 1 and ((-height |Q| — 2. 

automaton is leaktight, so the extended Markov monoid algorithm will find the value 1 
witness and correctly answers that it has value 1 . 
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4.5. Finding witnesses in the Markov monoid. In the subsection, we will prove the 
following complexity result. 

Proposition 4.12. There exists an algorithm which checks in polynomial space whether an 
automaton is leaktight and whether in such case it has value 1 . 

Following Theorem 13.31 checking whether a leaktight automaton has value 1 boils down 
to finding a value 1 witness in the Markov monoid. Similarly, following Theorem 14.11 
checking whether an automaton is not leaktight boils down to finding a leak witness in the 
extended Markov monoid. Note that in both cases, checking whether a given limit-word or 
extended limit-word is a witness is easily done in polynomial time. 

Since we aim at proving that those two tasks can be computed in PSPACE, which is 
closed under complementation, it suffices to show how to find a witness in the (extended) 
Markov monoid. For the sake of readability, we here only deal with the Markov monoid, 
but similar ideas apply to the extended Markov monoid. 

We describe an algorithm to guess a witness in the Markov monoid. The key property 
given by Theorem 14.71 is that we can restrict ourselves to at most \Q\ nested iteration 
operations. 

As the corresponding property was proved by Kirsten |Kir05j in the context of distance 
automata, also to obtain a PSPACE algorithm, the following algorithm is also an adaptation 
of [ |Kir05] . Rather than a formal proof, we here give an intuitive description of the algorithm. 

A witness can be described as a tree whose nodes are labelled by limit-words, of depth 
at most 2 • \Q\ + 1, as follows: 

• a leaf is labelled either by a for a G A or by 1, 

• an internal node can be a concatenation node, then it is labelled by u = vj • • • v^ for 
k < 2^ and has k children, labelled by vi,..., v^, 

• an internal node can be an iteration node, then it is labelled by and has one child 
labelled u. 

We describe an algorithm that guesses such a tree. It starts from the root, and travels over 
nodes in a depth-first way: from top to bottom (and up again) and from left to right. In 
a node, the algorithm stores the branch that leads to this node, and for each node in the 
branch the limit-word obtained by concatenating all the left siblings of this node. From 
a node, the algorithm guesses a limit-word, and whether it will be a leaf, a concatenation 
node or an iteration node. In the first case, it goes up and checks the consistency of this 
guess. In the two other cases, it updates the value of this node by concatenating the new 
guess with the previous value and goes down. 

Although the tree is of exponential size, in each step the algorithm only stores 2 • \Q \ +1 
limit-words at most, so it runs in polynomial space. 

5. Examples and subclasses of leaktight automata 

In this section, we investigate further the class of leaktight automata, by giving examples 
of leaktight automata, exhibiting subclasses, and showing closure properties. In particular, 
we prove that hierarchical automata, jj-acylic automata and simple automata are all strict 
subclasses of leaktight automata. (Actually, since jj-acylic automata are already a subclass 
of simple automata, we do not consider them.) This implies that our decidability result 
extends the decidability results from COP ) . ICT12) . 









PROBABILISTIC LEAKTIGHT AUTOMATA 


25 


5.1. Two basic examples. The automaton on figure [4] is leaktight. As we shall see, it is 
not hierarchical, nor simple, hence it witnesses that leaktight automata are not subsumed 
by hierarchical or simple automata. Its extended Markov monoid is depicted on the right- 
hand side. Each of the four directed graphs represents an extended limit-word (u, u+): if 
u (s,t) = 1 , then (s,t) is an edge, and if u(s,f) = 0 but u + (s,f) = 1 , then (s,t) is marked 
with +. 

The initial state of the automaton is state 0, and the unique final state is state 1. This 
automaton has value 1 and this can be checked using the extended Markov monoid: the 
two value 1 witnesses are a^ and b • a^. 



The automaton on figure [5] is leaktight. The initial state of the automaton is state 0, 
and the unique final state is state F. The Markov monoid has too many elements to be 
represented here. This automaton does not have value 1. 



Figure 5: A leaktight automaton which does not have value 1. 
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5.2. Some closure properties. In this subsection, we show some closure properties: syn¬ 
chronised product and composition with a deterministic transducer. As for Proposition 14.41 
the automata constructions reflect in algebraic constructions, allowing to use the character¬ 
ization with leak witnesses given by Theorem 14.11 

In this subsection, we will omit initial and final states, as we are only interested in 
preserving the leaktight property, which does not depend on it. 

Definition 5.1 (Synchronised product). Let A = and B = (Q B ,A B ) be two 

probabilistic automata. 

A synchronised product of A and B is: 

Ax B = {Q a x Q b , A), 
where A (q,a) = (A A (q,a),A B (q,a)). 

Proposition 5.2. The leaktight property is stable by synchronized product. 

Proof. The extended Markov monoid Q AxB of the synchronized product embeds into the 
direct product Q A x Q® of the extended Markov monoids of each automaton. 

Let (u, u_|_) be an extended limit-word in Q AxB . Define 114 ( 5 , t) = 1 if there exists 
s', t! £ Q b such that u((s, s'), ( t , t')) = 1, and similarly for u +i _ 4 , ug and u +j g. We have the 
following equivalence: 

u ((s,s'),(t,t')) = 1 4=> u A (s,t) = 1 Au B (s',t') = 1, 
and similarly for u + . 

Relying on this, we map (u, u+) £ Q AxB to ((u 4 ,u +v 4 ) , (ug,u +i g)). An easy induc¬ 
tion shows that this map is an embedding into Q+ x Q®. 

Consequently, the extended Markov monoid of the synchronised product contains a leak 
witness if and only if one of the extended Markov monoid contains a leak witness. □ 

The last closure property we prove will be useful in the next section. 

Definition 5.3 (Composition with a deterministic transducer). Let A = (Q^,A^-) be a 
probabilistic automaton, and Ai = (Q M , Q A , A-^) a deterministic transducer over A, i.e. 
A m : Q m x Q a -»■ Q m . 

The composition of A by M is: 

A®M = (Q A xQ M , A), 
where A(q,p,a) = {A A (q, a), A M (p, q)). 

Proposition 5.4. The leaktight property is stable by composition with a deterministic trans¬ 
ducer. 


Proof. Following the same reasoning as in 14.21 one can show that the extended Markov 
monoids for A and for A <S> A4 are isomorphic. □ 
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5.3. Leaktight automata strictly contain hierarchical automata. The class of hier¬ 
archical automata has been defined in [CSVllj . where it was proved that they are recognize 
exactly the class of w-regular languages. The states Q of a hierarchical automaton are 
sorted according to levels such that for each letter, at most one successor is at the same 
level and all others are at higher levels. 

Formally, there exists a mapping rank : Q —»• N such that for all a £ A, for all states 
s,t such that P^i(s —?■ t) > 0, we have rank(s) < rank(f). Furthermore, if P_ 4 (s -A t) > 0 
and P^(s —> t 7 ) > 0 but rank(s) = rank(t) = rank(t'), then t = t'. 

Proposition 5.5. Every hierarchical automata is leaktight. 

Proof. We prove by induction that for every extended limit-word (u, u_|_) in the extended 
Markov monoid of a hierarchical automaton, for every states s,t,t 

u_|_(s,f) = 1 rank(s) < rank(t), (5.1) 
u + (s,t) = 1 A u + (s,t / ) = 1 A rank(s) = rank(f) = rankfi 7 ) => t = t', (5.2) 

u(s,i) = 1 => rank(s) < rank(t), (5.3) 
u(s,f) = 1 A u(s,t') = 1 A rank(s) = rank(f) = rank(t ; ) ==> t = t!. (5.4) 

Note that m and (|5.2D imply (15.3p and (|5.4D . since u(s,t) = 1 implies u + (s,t) = 1. The 
key property following from (15.21) and (|5.3I) is that the recurrence classes of u and of u + are 
singletons. 

This is trivial for (1,1). The case of (a, a) is the definition of hierarchical automata. 
The induction step for concatenation is routinely checked, and trivial for the iteration. 

We now prove that the extended Markov monoid of a hierarchical automaton does not 
contain any leak witness. We prove a slightly stronger statement; for every extended limit- 
word (u, u + ) in the extended Markov monoid of a hierarchical automaton, for all states 
q,r: 

u +(r,q) = 1 A r is u-recurrent u(r, q) = 1. (5-5) 

This is clear for (1,1) and for (a, a). 

Concatenation. Let (u, u + ) and (v,v + ) be two extended limit-words satisfying (15.51) . 
Consider two states r, q such that (u + • v + )(r, q) = 1 and r is (u • v)-recurrent. 

Since r is (u-v)-recurrent, the recurrence class of r for (n- v) is r itself, so (u-v)(r, r) = 1 
and (u • v)(r,p) = 1 implies p = r. Let f be a state such that u(r, t) = 1 and v(t, r) = 1. 
Thanks to (15.31) . rank(f) = rank(r). 

We argue that n (r,p) = 1 implies p = t. Indeed, let p be a state such that u(r, p) = 1. 
There exists a state p' such that v(p,p') = 1, so in particular (u • v)(r,p') = 1, so p' = r. 
By (15.31) . rank(r) < rank(p) < rank(r), so they are equal. By (15.41) . since u(r, t) = 1, 
u(r,p) = 1 and rank(r) = rank(f) = rank(p), we have p = t. It follows that the state t is 
u-recurrent: by Lemma [2.21 there exists a state p such that u (r,p) = 1 and p is u-recurrent. 
The above remark implies that p = t. 

We argue that v(f,p) = 1 implies p = r. Indeed, let p be a state such that v(t,p ) = 1. 
We have (u- v)(r, p) = 1, so p = r. It follows that the state r is v-recurrent: by Lemma l2.21 
there exists a state p such that v(t,p) = 1 and p is u-recurrent. The above remark implies 
that p = r. 

Since (u + • v + )(r, q) = 1, there exists s G Q such that u + (r, s) = 1 and v + (s, q) = 1. By 
induction hypothesis for (u, u + ), since u + (r, s) = 1 and r is u-recurrent, we have u(r, s ) = 1. 
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Since r is u-recurrent, its recurrence class is r itself, so s = r. By induction hypothesis for 
(v,v_|_), since v + (r, q) = 1 and r is v-recurrent, we have v(r, g) = 1. 

It follows that (u • v)(r, q) = 1. 

The case of iteration is easy. □ 

The inclusion is strict, an example is given by figure [5J 

5.4. Leaktight automata strictly contain simple automata. The class of simple au¬ 
tomata has been defined in [0T12] , where it was proved that the value 1 problem is decidable 
for a subset of this class (namely, for structurally simple automata). In the following we show 
that the class of simple automata is strictly contained in the class of leaktight automata. 
We fix A a probabilistic automaton. 

Definition 5.6 (Jets). A jet is a sequence (J n )neN where Jt-CQ for each k £ N. 

Definition 5.7 (Simple process [CT12] ). Let w £ AN be an infinite word. The process 
induced by w from the state p is simple if there exists A > 0 and two jets (A n ) ne ^, (B n ) n& fq 
such that: 

(1) for all k £ N, A\ z and B & are disjoint and A/. U B^ = Q, 

(2) for all k £ N and all q £ A^, P _^(p w<k y q) > A, 

(3) limn^ooP^p ———> B n ) = 0. 

Definition 5.8 (Simple automata |CT12j l. A is simple if for every infinite word w and 
every state p the process induced by w from p is simple. 

Theorem 5.9. Every simple automaton is leaktight. 

The remainder of this subsection is devoted to the proof of Theorem 15.91 The proof is 
divided into two parts: first, we define non-simplicity witnesses, which are elements of the 
Markov monoid that witnesses the non-simplicity of an automaton, and second we show that 
if the Markov monoid of an automaton contains a leak, then it also contains a non-simplicity 
witness. 

5.4.1. Non-simplicity witness. 

Definition 5.10 (Non-simplicity witness). A triple (u, v,w) of elements of the Markov 
monoid is a non-simplicity witness if there exist states r, t such that: 

(1) uv*w is idempotent, 

(2) r is uv^w-recurrent, 

(3) uv(r,f) = 1, 

(4) t is v-transient. 

Proposition 5.11. If the Markov monoid of a probabilistic automaton contains a non¬ 
simplicity witness, then it is not simple. 

To prove Proposition 15.111 we rely on the following two lemmata: 

Lemma 5.12. Let u be an idempotent element of the Markov monoid, r be a state u- 
recurrent, and (u n )neN a sequence of words that reifies u. Then there exists a constant 
7 > 0 and a strictly increasing map h : N —> N such that for all n £ N, we have: 

u h( 0) "' u h(n— 1) 
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Proof. Since the sequence of words (un)neN reifies the limit-word u, for all states s, t, 
P_ 4 (s —A t ) converges and: 


u(s, t) = 1 


limP^s ^A t) > 0. 


(5.6) 


Define: 

A = - • min{limP^i(s ^A t) | u (s,t) = 1}. 

2 n 

Thanks to (j5.6f) . there exists an increasing map h : N —> N such that the following two 
conditions hold, for all states s,t and for all n £ N: 


if u(s,t) = 0 , then P_ 4 (s - h(n -\ t ) < 


IQI-2-+ 2 ' 


(5.7) 


if u (s,t) = 1, then P^t(s - h - n -y t ) > A. (5.8) 

We now use m and (|5.8() to prove the desired result, for 7 = |. 

Let w n = u h ^ U/j(i) • • -u h ( n _iy Denote the u-recurrence class of r by R = {q £ Q | 

u(r,q) = !}• We first bound the quantity P^(r —^ Q \ 7?). Note that for q £ R and t 
a state, the following holds: if u (q,t) = 1 then t £ R, i.e. R is not left while following 
transitions consistent with u. It follows that the probability to leave R from r while reading 

w n is smaller than Ylk=o — (l -> Q \ i?), which is smaller than ^ by (15.711 . Thus 

P_ 4 (r - n ~ 1 > R) > Now, since r is u-recurrent, and u is idempotent, for all q £ R we have 

u(q, r ) = 1, so using (15.81) we get that P^(g - h{ " S r) > A. It follows that P^(r ^A r) > 
which concludes. □ 


Lemma 5.13. Let w £ A u be an infinite word. If there exist states p, s,t, (i n )ne(jn)ne N 
and 7 > 0 such that: 

(1) for all n £ N, P^(p W<1 - "> s) > 7 , 

( 2 ) for all n £ N, i n < j n and P^t(s — ^ t ) > 0 , 

(3) lim^ooPy^p ^A t ) = 0, 

f/ien t/ie process induced by w from p is not simple. 


Proof. Assume towards contradiction that w induces a simple process from p with bound 
A. We first argue that for infinitely many n £ N, we have s £ Ai n and t £ Bj n . Indeed, if 
this is not the case, then for n £ N large enough either s (f Ai n or t Bj n , so either for 
infinitely many n £ N we have s Ai n , or for infinitely many n £ N we have t ^ Bj n . The 
first case is contradicted by (1), the second case by (3). 

Let n £ N such that s £ Ai n and t £ Bj n , since P^(s — > t) > 0, along a path from s 

to t there is a transition from the jet A to the jet B. Formally, there exists k n such that i n < 
k n < jn and the k n th transition goes from q kn £ A kn to q kn +1 £ B kri+ \. This transition is a 
one-step transition in the automaton A; denote by p m ; n the minimal non-zero probabilistic 

transition in A, we have P A (q kn w . [kn ' kn+ }\ q kn+1 ) > Pmin . Now, consider P A {p W - kn > q kri+ 1 ); 
since q kn £ A kn , we have P A (p W<k - n > Qk n ) > A, so P A (jp --A q kn +i) > A • p min . But for 
infinitely many n £ N, q kn+ \ £ B kn+1 , contradicting lim^oo P. 4 (p ^A B n ) = 0. □ 
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Now we are ready to prove Proposition 15.111 

Proof. Let (u, v, w) be a non-simplicity witness, and r, t € Q such that r is uv'w-recurrent, 
uv(r, t) = 1 and t is v-transient. Let (u n ) n& ^, (f n )neN) (R'n)neN be sequences of words which 
reify u, v, w respectively. 

Thanks to Proposition 12.111 there exists a strictly increasing map / : N —> N such that 
(uj( n )) ne ^ reifies v^. Note that since (^/(n))neN is a subsequence of (u n ) nS N, it also reifies 

v. Since v is idempotent, for all k € N, ('°/( n ))neN reifies v fc . By assumption uv(r, t) = 1, 

'U'n'vb,. 

so there exists Nk € N such that for all n > Nk, FU(r-- —> t) > 0. 

Let g(n) = max(n, N n ). Since g is increasing, (ug(n) v f^'l n ))Wg{ n )) n &i is a subsequence 
of (u n v r ^ r ^w n ) n (zf^. so it reifies uv^w as well. By definition of the function g, we have: 

for all n € N, P^(r “ g( " ) ^ ( ? ( " i ) > t) > 0. (5.9) 

We apply Lemma 15.121 to the limit-word uv^w, the state r and the sequence of words 
( u g(n)Vfj^ n ^w g ( n )) n e N, an d obtain h a strictly increasing map and a constant 7 > 0 . 

Define the new sequence of words {z n ) neN = (u g ^ n ^ ■ • w 9 (h( n ))) ne n, and 

(^rOneN = { u g(h(n )) ‘ v f(g(h(n )))^ n ^ 1 have: 

for all n G N, P_ 4 (r Z -■ — ~ - 1 > r) > 7 . (5.10) 

Let z = Z 0 Z 1 • • •. We argue that the conditions of Lemma f5.131 are met: 

( 1 ) for all n € N, P^(r "° - Zn ~ 1 > r) > 7 , 

( 2 ) for all n G N, P^(r t) > 0 , 

(3) lim MOO P^(r ' • '••• • '' t ) = 0, 

The item (1) is (15.101) . the item (2) follows from (15.91) . so we consider (3). 

First, note that ( x n ) ne ^ reifies uv p . Indeed, we first argue that (' u /( 9 („)))neN reihes 

v^: it follows from Proposition 12.111 since / og:N^Nisa strictly increasing map 
satisfying / o g > f, and that ('Wj( n) )neN reifies v 1 *. Now, ( v f(g(h( n )))^ N is a subsequence 
of ( v f( g ( n ))) n eN, so it reifies as well, and it follows that (x n ) ne n reifies uvl 

Second, let q £ Q, since t is v-transient, u v^(q,t) = 0. Since ( x n ) n reifies uv*, we 
have limn^oo P_ 4 (g ^ — 0 . Now, 

V A (r () = £ F ^ r 9) • P A <1 ^ *)• 

q&Q 


and for each term in the sum, the second factor converges to zero, so lin^^oo Pyi(r Zn i x _ n y 
t ) = 0 . 

Thus Lemma 15.131 applies, and z induces a non-simple process from p, so A is not 
simple. □ 
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5.4.2. The presence of a leak implies the presence of a non-simplicity witness. Now we show 
that the presence of a leak witness implies the presence of a non-simplicity witness. 

Proposition 5.14. If the extended Markov monoid of a probabilistic automaton contains 
a leak witness, then it also contains a non-simplicity witness. 

In the following proof, we will make use of the notion of {(-height for an extended limit- 
word. The ((-height of a limit-word was defined in Section 14.61 the ((-height of an extended 
limit-word (u, u + ) is the ((-height of u. 

Proof. Let A be a probabilistic automaton whose extended Markov monoid contains a leak 
witness. Consider the subset C of extended limit-words (u, u + ) in the extended Markov 
monoid such that there exist states r , t satisfying: 

(1) (u, u + ) is idempotent, 

(2) r is u-recurrent, 

(3) u (r,t) = 0, 

(4) u +(r,t) = 1. 

Note that C is non-empty since every leak witness is in C. 

Consider an element (z, z + ) in C of minimal ((-height and let r,t G Q such that r 
is z-recurrent, z (r,t) = 0 and z +(r,t) = 1. In particular, we have z ^ z+, so in any 
decomposition of (z, z+) into concatenation and iteration there must be at least one iteration. 
Consequently, z = uv*w for some u, v, w, and uvw has a strictly smaller ((-height than z. 

Let R be the z-recurrence class of r, and T the set of v-transient states. We argue that 
the following holds: 

there exist r' € R and t! € T such that u v(r',t') = 1. (5-11) 

Assume towards contradiction that (15.111) does not hold, i.e. for all r' € R and t' S T, we 
have uv(r',i') = 0, then we prove that (uvw, z + )l^+l ! is in C, contradicting the minimality 
of (z,z+) as it has strictly smaller ((-height. 

First observe that for all states q we have uv(r / ,q) = uv^(r ; ,g), which implies that for 
all r' € R and state q, we have: 

uvw(/, q) = uv*w(r / , q) = z(/, q). (5-12) 

We check that (uvw,z + )l^+l ! is in C, with the states r,t as witnesses: 

(1) (uvw,z + )ie+l ! is idempotent, this follows from Lemma 12.21 

(2) r is (uvw)l^+l ! -recurrent. Indeed, let q E Q such that (uvw )l®+l ! (r, q) = 1. It follows 
that (uv^w)l^+l ! (r, q) = 1, but (uv#w)^+l ! = z^+l ! = z, so z (r,q) = 1. Since r is 
z-recurrent, we have z (q,r) = 1 and q € R, so (|5.12j) implies that z (q,r) = uvw (q,r), 
thus uvw(g, r) = 1. Also, z (r,r) = uvw(r, r), so uvw(r, r) = 1, and altogether 
(uvw)l 0 +l ! (q, r) = 1. 

(3) (uvw)l®+l ! (r, t) = 0. Indeed, assume towards contradiction that (uvw)^+l ! (r, t) = 1. 
Then there exist qo,qi, ■ ■ ■ M\g + \\ such that qo = r, for i € {0,..., \G+\\ — 1} we have 
uvw ((ft, qi+i) = 1 and q\g + \\ = t. We prove by induction on i € {0,... , |£/+|! — 1} that 
qi is in R and that z(qi, q t +i) = 1. Assume qi is in R , then by (15.12|) . uvw((/j, qi+\) = 
z(qi,qi + i), so z(qi,qi + \) = 1. But qi is in R, which is a recurrence class for z, so qt + 1 
is in R as well, concluding the induction. Thus, we have z^+l '(r,t) = 1, and since z is 
idempotent z(r,t) = 1, a contradiction. 

(4) z^ + '(r, t) = 1. It follows from the fact the z + is idempotent, and that z +(r,t) = 1. 
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We reached a contradiction, since (uvw,z+)^+l ! has a strictly smaller {(-height than 
(z,z_|_). It follows from (|5.11l) that (u, v, w) is a non-simplicity witness for the states r' and 
t', concluding the proof. □ 

The proof of Theorem 15.91 is now a simple combination of Proposition 15.111 and Propo¬ 
sition [5J3J The inclusion is strict: figure [4] provides an example of a leaktight automaton 
which is not simple. 


6. Probabilistic w-automata 

In this section, we relate the value 1 problem for probabilistic automata over finite words 
and over infinite words, as introduced and studied in [ BBG12 ], We state a general theorem, 
showing the equivalence of the value 1 problem for automata over finite words and the value 
1 problem for automata over infinite words with the parity condition. This theorem allows 
to extend the decidability results from finite words to infinite ones. 

For the definitions of probabilistic automata over infinite words, we refer to |BBG12j . 
For the sake of readability, we introduce two notations in this section. First, we denote 
by P. 4 ,w{E) the probability of the measurable event E when reading the infinite word w 
on A. Second, we denote by P^(u) the probability that the finite word u is accepted by 
A with 5 as initial distribution. Note that in our definition, probabilistic automata have 
a unique initial state; however here we need to deal with automata having general initial 
distributions, so we sometimes consider the more general tuples A = ( Q , 5, A, F ) where 5 is 
the initial probability distribution. 

Theorem 6.1. Let A = (Q,So, A,c) be a probabilistic parity automaton where c : Q ^ N 
is a priority function. Consider the deterministic transducer Ai over A, which keeps track 
of the minimal priority seen: Ai = (c(Q),Q^, A_m) where Aj^{q,c) = min(c, c(q)). 

The automaton A over infinite words has value 1 if and only if there exists R C Q, such 
that the two following probabilistic automata over finite words have value 1: 

• The automaton A(R) with R as set of final states; 

• The automaton A <8> Ai with the uniform distribution over R c = {(q,c(q)) \ q £ R} as 
initial distribution and {(q,e) \ q £ R and e even} as set of final states. 

Before giving the proof, we need two lemma. 

Lemma 6.2. Let A = (Q, 5, A, F) be a probabilistic automaton, u a word and e > 0. Let 
p, = min{<5(q) | q £ Supp(<5)}. IfF s A (u) > 1 — £ ■ p, then for all q £ Supp(<5), P^(u) > 1 — e. 

Proof. 

<5 (^)' IP ^( u ) ^ l-e-fi. 

geSupp(<5) 

Let q £ Supp(<5), since the probabilities are bounded by 1: 

5{q) ■ P q A (u) + Y S (p) ^ 1-e-li. 

peSupp(<5), p^q 

'-v-' 

l-5(q) 

Hence: 

p » ^ a 
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□ 

Corollary 6.3. Let A = (Q, 5, A, F) be a probabilistic automaton. If A has value 1, then: 

• For all distributions 6' such that Supp(d') C Supp(<5), it has value 1 with 5' as initial 
distribution. 

• For all distributions 5' such that X^eSupp(<5) ^(h) — ^ ^ as value at least 9 with 5' as 

initial distribution. 

Proof. Assume A has value 1. Let e > 0, then there exists a word u such that P^it) > 1—£•//. 
Thanks to Lemma 16.21 this implies that for all q € Supp(h), we have P^(tt) > 1 — e. 

Now for 5': 

> E > ( E ^(?)] ■(!-£)• 

<jeQ ?eSupp(5) y<jeSupp(<5) J 

For the first item, note that if Supp(<5') C Supp(<5), then X]qeSupp(<5) ^'(?) = 1) so P^(^) — 
1 — e, so A has value 1 with 6' as initial distribution. 

The second item follows from the last inequality, implying P^(u) > 9 ■ (1 — e), so A has 
value at least 9 with 5' as initial distribution. □ 

Lemma 6.4. For all £ > 0, there exists a sequence (£k)k> o satisfying: 

(1) For all k >0, we have 0 < £k < 1; 

(2) rifc>o ~ £ > 

(3) For all k >0, we have Yl r <k £ p > £ k+i- 

Proof. Define v k = 1 — which clearly satisfies 1. It also satisfies 3.: 

rip<fc u p = n P < k ~ 2 2p 1 

n P<fc 2 2P -i 

22 k + 1 -l 

The inequality n p <fc v p > v k +1 is equivalent to: 

2- (np 2 '- 1 )) <2 2 * +i -i, 

which is easily proved by induction, since (2 2fc+1 — l) 2 < 2 2k+2 — 1. 

The infinite product ri/, >o u k has a value u ~ 0.350184, in particular 0 < v < 1. Let 

£ > 0, then for A = , which satisfies A > 0, the sequence (z^)fceN satisfies the three 

conditions. □ 

We are now fully equipped for the proof of Theorem 16.11 

We begin with the left-to-right direction. Let (u; n ) ng M be a sequence of infinite words 
such that P^ jlUn (Parity(c)) converges to 1. For each n, denote by (<5k)fceN the sequence 
of distributions assumed by A when reading w n . Since there are finitely many possible 
supports, there exists a subsequence where all distributions have the same support. Since 
[0,1]^ is a compact space, we can extract from this subsequence another one, denoted by 
(S'f ( k j)k£ n, which converges to a distribution denoted by 5 n . By the same compactness 
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argument, at the expense of considering a subsequence of ( w n ) n we can assume that 
(<5 n ) nS N converges to a distribution <5, whose support is denoted by R. 

We now prove that A(R) as well as A <8> M have value 1. 

The automaton A(R). Let e > 0. Since (<5 n ) n eN converges to 5 , for n large enough, 
we have ||<5 n — 5||oo < 2 Tr\- We n l ar § e enough; since (^"„(fc))fceN converges to 5 n , for k 
large enough, we have 1 |< 5 ”' ^ — (5 n ||oo < 2 ^r\- We suc ^ a large k, altogether this implies 

W S U) - 

Now consider u the prehx of w n until position cj> n (k). We have: 



It follows that A(R) has value 1. 

The automaton A®A4. Let v = min{<5(</) | q S Supp(d)}, clearly 0 < v < 1. Since 
(d n ) ne N converges to 5, we can assume that for all n > 0, ||d n — <5||oo < |, considering the 
sequence (u> n )neN from some index on. For each n, the sequence (^(j.jJfcgN converges to 
5 n , so again by considering the sequence from some index on, we can assume that for all 
k > 0, || 5^ ^ — d n ||oo < j, hence || 5^ — (5||oo < f • As a result, for all q € R, we have 
6 l n (k)^) > § > 0, so R = Supp(<5) C Supp(5 n ) C Supp(^ n(fc) ). 

Let e > 0, and n such that P_ 4 )l( , n (Parity(c)) > 1 — £ ■ j. 

Let Biichi(d) be the set of runs where the color d is reached infinitely often and by 
CoBiichi(< d ) the set of runs where colors less than d are reached only finitely often. Observe 
that: 

Parity(c) = Biichi(d) fl CoBiichi(< d). 

d even color 

Let Safe(< d, k ) be the set of runs where colors less than d are not reached anymore 
after the position <j> n (k). Observe that: 

Biichi(d) O CoBiichi(< d) C Biichi(d) 0 |^J Safe(< d,k), 

k>0 
so: 

P. 4 jU , n (Bhchi(d) 0 CoBtichi(< d)) < ^ a , w „ I Btichi(d) fl |^J Safe(< d,k) 

\ fc>o 

Since the sequence Btichi(d) fl (Safe(< d,k))k>o I s increasing with respect to set inclusion, 
for some k we have: 

P>t,«;„(Buchi(d) O Safe(< d, k)) > P_ 4 ,w n (Biichi(d) 0 CoBtichi(< d)) — e ■ —. 

Note that here, k depends on d; however, since if it holds for k then it holds for any bigger 
k and that there are finitely many even colors d, we can assume that k is uniform over all 
even colors d. 

Now, let Reach(d, k, k') be the set of runs where the color d is reached between the 
positions <j) n (k ) and cj) n (k'). Observe that: 

Biichi(d) 0 Safe(< d, /c) C Reach(d, k, k') 0 Safe(< d, k), 

k’>k 
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so: 

P^ u , n (Biichi(G?) fl Safe(< d, k)) < ^A,w n ( Reach(<i, k, k') n Safe(< d. k) 

\k’>k 

Since the sequence (Reach (d,k,k') fl Safe(< d,k)) k , >k is increasing with respect to set in¬ 
clusion, for some k' we have: 

P. 4 ,(Reach (d, &0 H Safe(< d, k)) > P v 4 i „, n (Biichi(d) fl Safe(< d, k)) — e ■ —. 

Here again, k' depends on d; however with the same reasoning as above, we can assume 
that k! is uniform over all even colors d. 

Altogether, this implies: 

(Reach(d, k, k') n Safe(< d, k)) > P_ 4 ,«j„(Biichi(d) fl CoBtichi(< d)) — £ ■ —, 
and summing these equalities for each even color d: 

P. 4 ,™r, ( p arity(c, k, k')) > even color P M,^(Reach (d, k, k') n Safe(< d, k)) 

> Pm, rcn (Parity(c)) - e • 

where Parity(c, k, k') is the set of runs where the minimal color seen between the positions 
4> n (k) and cj> n (k') is even. 

Let v be the infix of w n between the positions (f> n (k ) and 4> n (k '). The distribution 
^ is over the set of states Q\ we embed it as 6 over the set of states Q x c(Q), set¬ 
ting 5(q,c(q)) = 6^ ^ k Jq ) and 0 otherwise. By construction of v, we have P^a-i^) = 
PM,«)n(P ar ity( c ) k, k')) > 1 — £■%. However, the initial distribution of A <&M. is uniform over 
R c . We apply Lemma HOI by construction fi > |, so P M<g).M ( u ) > 1 — e ■ fi. It follows that for 

all q € Supp(<5" n(0) ), P q A®M( v ) > 1 ~ e - Since R C Supp (<5£ n(fc) ), we have Pm®m(^) > 1 - 
hence A <8) M has value 1. 



We now prove the right-to-left direction. Let R such that both A{R) and A®M. have 
value 1. Let e > 0, consider the sequence (£k)k >o given by Lemma 16.41 

Since A(R) has value 1, there exists u such that P^(ij)(w) > £o- We show the existence 
of a sequence w\, W 2 , ■ ■ ■ of words such that for all k > 0: 

¥ A(R)( u - w i ■ ■■■ ' w k ) > and p %_m(^+i) > e fc+i> 

p<k 

where 5k is the distribution obtained by reading u ■ w\ ■ W 2 • ■ ■ ■ ■ Wk on A; the distribution 
5k is over the set of states Q, we embed it as 5 k over the set of states Q x c(Q), setting 
5 k (q,c(q )) = 5k(q) and 0 otherwise. 

We proceed inductively; assume w\,... ,Wk have been chosen. Note that by construc¬ 
tion, 5k{q) > Y\ p <k e p- However, the support of 5k is not included in R , so the first 

item of Corollary 16.31 does not apply. Then r c <%(<?) > Y\ v <k £ p- 

Since A <g> M has value 1, thanks to the second item of Corollary 16.31 it has value at 
least FI P <k £ p f° r ^k as initial distribution. Together with 3., this implies that there exists 

w k+ i such that P^^Wfc+i) > s k +i- 
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We have: 

^A(R) i u ■ W\ ■ W 2 ■ ■ ■ ■ ■ w k+ 1 ) > P A ( R ) {u ■ wx ■ W 2 ■ ■ ■ ■ ■ w k ) ■ P^ (i?) Ofc+l) 

> ^A(R)(u ■ m ■ W2 ■ ■ ■ ■ ■ W k ) • P^_ M (w;fc +1 ) 

— (rip<fc e p) ■ £ k+i 
= rip<fc+i £ pi 

which concludes the inductive construction. 

Let w = u ■ w\ ■ u >2 ■ ■ ■ ■■ We evaluate P^ iW (Parity (c)): 

P-4,to (Parity (c)) > P^ (fl) (u) ■ JJ Pj^^fc+i) > £o • JJ £fc = II £k - 1 _ e ' 

fc>0 k >1 k >0 

It follows that A has value 1 as a probabilistic parity automaton, which concludes the proof 
of Theorem 16.11 

Corollary 6.5. The value 1 problem for leaktight automata over infinite words with the 
parity condition is decidable and PSPACE -complete. 

Indeed, observe that if A is leaktight, then by Proposition 15.41 so is A® M.. 
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Conclusion 

We introduced a subclass of probabilistic automata, called leaktight automata, for which 
we proved that the value 1 problem is PSPACE-complete. This subclass generalizes all 
subclasses of probabilistic automata whose value 1 problem is known to be decidable. 

A challenging perspective is now to find subclasses of partially observable Markov deci¬ 
sion processes where the value 1 problem is decidable (some preliminary results were given 
in jGQ14j ). and to extend our results to the setting of partially observable stochastic games, 
which is even more challenging. 
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